Multi-source image fusion

ABSTRACT

A method can include registering a first image of a region to a three-dimensional (3D) point set of the region to generate a registered first image, registering a second image of the region to the 3D point set to generate a registered second image, identifying, based on the 3D point set, geometric tie points of the registered first image and the registered second image, projecting, using an affine transformation determined based on the identified geometric tie points, pixels of the registered first image to an image space of the registered second image to generate a registered and transformed first image, and displaying the registered and transformed first image and the registered second image simultaneously.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims the benefit of U.S. Provisional PatentApplication No. 62/696,107, filed Jul. 10, 2018, entitled “IMAGEREGISTRATION TO A 3D POINT SET”, which is incorporated by referenceherein in its entirety.

TECHNICAL FIELD

Embodiments discussed herein regard devices, systems, and methods forfusing image pixels of images from same or different types of (e.g.,sensors). Embodiments can provide fusion without voids.

BACKGROUND

In certain applications, images from different sources may be processedin a manner to fuse (e.g., combine, overlay, augment, or the like) suchimages. In some cases, one image may be projected through a point cloudto the other image for such fusion. However, such a projection can leavevoids in the resulting fused image. For some applications, the presenceof voids is unacceptable. Thus, it can be advantageous to facilitatefusing image chips from two or more sources without voids.

SUMMARY

Image fusion may include registering images from disparate sensor typesto a high-resolution three-dimensional (3D) point cloud and updating thegeometries of both images to match the 3D point cloud. This allows oneimage to be projected through the point cloud to the other image to fusethem. However, such a projection may leave voids. At least some of thevoids can be due to occlusions in the image being projected and at leastsome of voids can be due to voids in the point cloud itself. For someapplications, the presence of voids is unacceptable. For theseapplications it can be preferable to have less accurate fusion than tohave voids. Embodiments herein can fuse images without voids while stilltaking advantage of a high accuracy registration to a 3D point cloud.

Various embodiments generally relate to devices, systems, and methodsfor fusing image data of images from different sources, such as imagesfrom two or more sensors. In some embodiments, the techniques describedherein may be used for fusing image data of images from the same source,(e.g., two images from the same sensor). The image fusion can be ofimage data corresponding to a same or substantially same region, such asa geographical region, a building, a vehicle (e.g., a manned or unmannedvehicle), or other object, or the like. In accordance with one aspect ofthe disclosure, an example method for fusing images comprises:registering a first image of a region to a 3D point set of the region togenerate a registered first image, registering a second image of theregion to the 3D point set to generate a registered second image;identifying, based on the 3D point set, geometric tie points of theregistered first image and the registered second image; and projecting,using an affine transformation determined based on the identifiedgeometric tie points, pixels of the registered first image to an imagespace of the registered second image to generate a registered andtransformed first image. In some embodiments, the method furthercomprises displaying the registered and transformed first image and theregistered second image superimposed.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralscan describe similar components in different views. Like numerals havingdifferent letter suffixes can represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments or examples discussed inthe present document.

FIG. 1 illustrates, by way of example, a diagram of an embodiment of anelectro-optical (EO) image of the Capitol Building area of Washington,D.C.

FIG. 2 illustrates, by way of example, a diagram of an embodiment of asynthetic-aperture radar (SAR) image of the Capitol Building area inWashington D.C.

FIG. 3 illustrates, by way of example, a diagram of an embodiment of amethod for fusing images.

FIGS. 4 and 5 illustrate, by way of example, respective diagrams ofimages with geometric tie points.

FIG. 6 illustrates, by way of example, a diagram of an embodiment of animage after registration and transformation.

FIG. 7 illustrates, by way of example, a diagram of an embodiment of animage after registration.

FIG. 8 illustrates, by way of example, a diagram of an embodiment of animage after the method is performed on the first image and the secondimage.

FIG. 9 illustrates, by way of example, a block diagram of an embodimentof a machine on which one or more of the methods, such as thosediscussed about FIGS. 1-8 can be implemented.

DETAILED DESCRIPTION

A portion of the disclosure of this patent document contains materialwhich may be subject to (copyright or mask work) protection. The(copyright or mask work) owner has no objection to the facsimilereproduction by anyone of the patent document or the patent disclosure,as it appears in the Patent and Trademark Office patent file or records,but otherwise reserves all (copyright or mask work) rights whatsoever.FIG. 1 is an image provided using data retrieved from DC Data Catalog(http://data.dc.gov/). FIG. 2 is an image provided courtesy of SandiaNational Laboratories, Radar ISR.

Embodiments generally relate to devices, systems, and methods for fusingimage data of images from different sources. The image fusion can be ofimage data corresponding to a same region, such as a geographicalregion, a building, a vehicle (e.g., a manned or unmanned vehicle), orother object, or the like.

A passive three-dimensional (P3D) point set can be extracted fromimagery. The passive 3D point set models the surface of the visiblescene including 3D structures, such as portions of buildings (e.g.,walls, windows, roofs, etc.), and vegetation (e.g., trees, shrubs,grass, etc.). Other elevation data can be generated of a samegeographical area. Elevation data can be from a digital elevation model(DEM) or other 3D point set. A difference between the DEM and other 3Dpoint sets can include the DEM having a regular spacing between points,while other 3D point sets have irregular spacing between points.

Bare earth elevation alters a surface 3D point set so that, when thebare earth points are viewed, the image models a view of a geographiclocation as if buildings and trees or other foliage were not there. Asurface elevation data set does not include such an alteration. A 3Dpoint set, very likely includes voids that are caused by the data usedto generate the 3D point set.

Embodiments herein do not introduce voids when projecting one imagespace to another image space. Embodiments can avoid voids by using atransformation (an affine transformation) that transforms one imagespace to the other image space, such as at a specific elevation (e.g., aground elevation at the center of the image space, an average elevationover a range of points in the 3D point set, a percentile of theelevation distribution of the points in the 3D point set, or the like).Embodiments herein provide a process for creating the affinetransformation and applying it to the images to be fused.

FIG. 1 illustrates, by way of example, a diagram of an embodiment of anelectro-optical (EO) image 100 of the Capitol Building area ofWashington, D.C. FIG. 2 illustrates, by way of example, a diagram of anembodiment of a synthetic-aperture radar (SAR) image 200 of the CapitolBuilding area in Washington D.C. The images 100, 200 are collected fromdifferent perspectives. The EO image 100 was captured from almostdirectly overhead relative to the Capitol Building and the SAR image wascollected from a more southerly perspective relative to the CapitolBuilding. These example images 100, 200 will be used to illustrate theprocess of fusing images using embodiments. It should be appreciatedthat EO and SAR are just two types of images used for illustrationpurposes to which embodiments apply. Other types of images, such asmulti-spectral, infrared (IR), or other image types that may be capturedusing other sensor types can be used in place of the EO or SAR image.

FIG. 3 illustrates, by way of example, a diagram of an embodiment of amethod 300 for aligning images, such as for subsequent image fusion. Themethod 300 as illustrated includes receiving or retrieving a firstimage, second image, and a 3D point set of a geographic location, atoperation 302; registering the first image to the 3D point set to updatethe geometry of the first image and registering the second image to the3D point set to update the geometry of the second image, at operation304; identifying, based on the 3D point set, an elevation for a grid ofgeometric tie points in the first and second registered images, atoperation 306; projecting the grid of geometric tie points to the firstand second registered images, respectively, to determine tie pointcoordinates 308; determining an affine transformation between thedetermined tie point coordinates of the first image and the determinedtie point coordinates of the second image, at operation 310; applyingthe affine transformation to pixels of the registered first image, atoperation 312; and displaying the transformed and registered first imagesuperimposed with (on or under) the registered second image, atoperation 314.

Examples of the first image and second image that can be received orretrieved at operation 302 are provided in FIGS. 1 and 2, respectively.As previously discussed the first image 100 and the second image 200 caninclude a same or different type of image. Types of images can include acolor image, grayscale image, infrared image, SAR image, multi-spectralimage, among others.

The 3D point set can include a digital elevation model (DEM), digitalsurface model (DSM), digital terrain model (DTM), bare earth DEM, orother 3D point cloud of a geographic area or other region or object. The3D point set can include latitude, longitude, and elevation data foreach pixel (sometimes called point) in the 3D point set or can includedata in a Universal Transverse Mercator (UTM) coordinate system.

The operation 304 can include aligning two or more images of a samescene with reference to a particular image. In the example of FIG. 1,the first image 100 and the second image 200 are registered,individually, to the 3D point set. The images can be captured from sameor different sensors, such as at same or different times, or at same ordifferent view-points (e.g., imaging perspectives). There are manytechniques for performing operation 304. Image registration techniquesinclude intensity-based and feature-based registration techniques.Source images, the first image 100 and the second image 200, are alignedwith a reference image, the 3D point set. Intensity-based registrationcompares intensity patterns, usually using correlation metrics.Feature-based techniques find correspondence between unique features(e.g., points, lines, contours, objects, or the like) of the images.Some image registration techniques use a combination of both intensityand feature based techniques.

Some registration techniques are linear, while others are non-linear(sometimes called “elastic” or “non-rigid”). Examples of lineartransformations include affine transformations. Examples of non-lineartransformations include radial-basis transforms, physical continuummodels, and large deformation models. Some transformations can beperformed in the spatial domain, while others can be performed in thefrequency domain.

A registration technique, using ground control points (GCPs) (e.g., aknown ground location and a known image location for that groundlocation) can be used to perform operation 304. From all the extractedGCPs, the imaging geometry of the image can be updated to match theGCPs. Adjusting the geometry of the image is now summarized. Imagemetadata can include an estimate of the camera location and orientationat the time the image was collected, along with camera parameters, suchas focal length. If the metadata was perfectly consistent with the 3Dpoint cloud, then every 3D point would project exactly to the correctspot in the image. For example, the base of a flag pole in the 3D pointset would project exactly to where one sees the base of the flag pole inthe image. But, in reality, there are inaccuracies in the metadata ofthe image. If the estimate of the camera position is off a little, or ifthe estimated camera orientation is not quite right, then the 3D groundpoint representing the base of the flag pole will not project exactly tothe pixel of the base in the image. But with the adjusted geometry thebase of the flag will project very closely to where the base is in theimage. The result of the registration is adjusted geometry for eachimage. Since both images 100 and 200 are both adjusted to the same 3Dpoint set, their geometries are consistent with each other. Again, anyregistration process can be used that results in an adjusted geometryfor each image being consistent with the 3D point cloud. The updatedgeometries resulting in the registered images facilitate alignment ofthe two images.

After each image 100, 200 has been registered to the 3D point set, theregistered first image could be projected through the 3D point set tothe space of the second image. Locations that are visible from theimaging perspective of the registered first image will fuse correctlywith the registered second image even for locations that fall in ashadow of the second image 200. However, there are locations visible inthe second image 200 that are occluded in the first image 100. Forexample, since the image 100 is viewed from almost directly overhead,the south facing walls seen by the SAR sensor in image 200 are notvisible in image 100. And if the image 100 had been collected from anoff-nadir perspective the ground on opposite sides of the buildingswould be occluded in image 100, creating a void when projected throughthe 3D point cloud into the space of image 200.

Because the collection geometry of the second image 200 is from asoutherly perspective, the building walls and surfaces on the south sideof the Capitol Building are not occluded from the imaging perspective ofthe second image 200. When the image 100 is projected to the image spaceof the second image 200, there are no valid pixels (corresponding to theoccluded locations) in the first image 100 from which to populate theimage in the projected space. The pixels in these areas are voids. Thesame would be true if image 100 had been collected from a differentperspective, say from the west. The ground on the east side of theCapitol Building would be visible in image 200 but occluded in image 100resulting in void areas when the image 100 pixels are projected to image200 space. As previously discussed, such voids can be unacceptable insome applications.

Similarly, voids in the 3D point set (places with no 3D knowledge) alsointroduce voids in the projected image. In cases where voids are notacceptable, an interesting question that emerges sometimes is whatintensities to use for the void pixels. One option is to make themblack. However, this may be problematic because the black regions in theprojected image may be mistaken for shadows rather than voids. Anotheroption is to mark them with a color that does not occur frequently inimages (either in natural or man-made scenes), such as bright purple, sothat the void areas are obvious to an image analyst. This technique canbe acceptable for human exploitation of the fused image, but can causeproblems for a machine learning algorithm that has not been programmedor trained to handle the voids.

As previously discussed, there are applications of image fusion, bothhuman exploitation and automated technique, in which void pixels areundesirable, so much so that an increased pixel fusion imprecision canbe preferred over voids. Therefore, techniques to transform one image toan image space of another image can be desired. In some embodiments, thetransformation can be accomplished using an affine transform asdescribed herein, rather than geometric projection through the 3D pointcloud to another image space.

Another technique of performing operation 304 can include locatingcontrol points (CPs) in the first image 100 and the second image 200.The CPs can be derived from the 3D point set. e The CPs can be used toperform a geometric bundle adjustment that brings the geometry of thefirst image 100 and the second image 200 into alignment with the 3Dpoint set (and therefore with each other). The result is that anyphysical 3D point in the 3D point set projects accurately to acorresponding pixel in each of the first image 100 and the second image200. However, as mentioned previously, the projection process can resultin void pixels when the perspectives of the two images are different orif there are voids in the 3D point set. Thus, a different process oftransforming an image space of the first image to an image space of thesecond image (or vice versa) is desired.

At operation 306, the 3D point set can be used to establish geometrictie points. A tie point is a four-tuple (row 1, column 1, row 2, column2) that indicates a row and column of the first image (row 1, column 1)that maps to a corresponding row and column of the second image (row 2,column 2). The geometric tie points can be determined based on anelevation about or around a center of the image 100, 200 (or portion ofthe image 100, 200) to be transformed. In some embodiments, thiselevation can be the elevation of the nearest 3D point to the center ofthe 3D point set. In other embodiments, the elevation can be theminimum, median, mode, average, or other elevation as calculated over arange of points in the image 100, 200 or a portion thereof. In general,the geometric tie points can be an elevation that approximatelyrepresents ground level near a center of overlap of the two images (orthe region of interest for the fusion). The operation can includehistogramming the elevation (sometimes called “Z”) values of the P3Dpoints in an area near the center of the region of interest and using anelevation at some percentile of the histogrammed elevations. Thepercentile can be the median, (e.g., 50^(th) percentile) or some lowerpercentile. Passive 3D (P3D) is 3D points derived from imagery, asopposed to 3D from active sources, such as LiDAR. Either can be used inembodiments.

The operations 306 and 308 can be for generating an affinetransformation at operation 310 to be applied to the registered firstimage at operation 312. The operation 308 can include laying out aregular grid of points in x, y ground space at the elevation identifiedat operation 306 (see FIGS. 4 and 5). Each point in the grid can beprojected to its corresponding image using the adjusted geometry of eachimage at operation 308. Each projected point of the grid is a geometrictie point. Corresponding geometric tie points in the first and secondimages form a geometric tie point coordinate.

FIGS. 4 and 5 illustrate, by way of example, diagrams of images 400 and500, respectively. The images 400 and 500 include the first image 100and second image 200 after registration and with identified geometrictie points 402 and 502. The geometric tie points 402, 502 can be used toestablish tie points between pixels of the first image 100 and pixels ofthe second image 200. The set of geometric tie points 402 and 502 can bea regular grid of points at the identified elevation.

The operation 310 can include determining the affine transformation thatmaps the tie points of the first image to the tie points of the secondimage. Sometimes the tie points are called geometric tie points. Atoperation 312, the affine transformation determined at operation 310 canbe applied to the first image and use that affine to transform the firstimage to the space of the second image.

At operation 310, an affine transformation between the two images can beidentified or determined, such as based on the geometric tie points. Theaffine transformation can be determined using a least squares fit to thegeometric tie points between the first image 100 and the second image200. One of the images 100, 200 can then be transformed to the imagespace of the other image, such as by applying the identified ordetermined affine transformation to each pixel. The result of the affinetransformation indicates the pixel in the other image corresponding to agiven pixel in a source image. Since the affine transformation ofoperation 310 does not project a destination pixel exactly to the centerof a pixel in the source image, bilinear, or other interpolation, can beapplied to the intensities of the 2×2 set of pixels nearest theprojected location, such as to identify the proper pixel location in thesource image.

An affine transformation is a linear mapping that preserves points,straight lines, planes. That is, parallel lines in a source image remainparallel after an affine transformation to a destination image.Different affine transformations include translation, scale, shear, androtation. The affine transformation of operation 310 translates alocation in the first image 100 to a location in the second image 200based on a projected location of geometric tie points 402 in the firstimage 400 relative to a corresponding projected location of geometrictie points 502 in the second image 500.

FIG. 6 illustrates, by way of example, a diagram of an embodiment of animage 600 after registration and transformation. The image 600 is thefirst image 100 after the operations 302, 304, 306, 308, 310, and 312have been performed. FIG. 7 illustrates, by way of example, a diagram ofan embodiment of an image 200.

FIG. 8 illustrates, by way of example, a diagram of an embodiment of animage 800 after the method 300 is performed using the first image 100and the second image 200. The image 800 includes the images 600 and 200superimposed on each other. Note that there are no voids in thetransformed first image and that the two images fuse well even thoughthe transformation was generated at a single elevation plane.

Because the transformation does not depend on the 3D point cloud, otherthan establishing the center elevation and for registration, voids inthe 3D point cloud do not result in voids in the transformed image.Further, occlusions do not introduce voids because there will always bean interpolated pixel intensity to fill each pixel in the destinationimage. The output is a fused pair of images with no voids in thetransformed image.

The synthetic image generation technique was tested on more than onehundred (100) images from sixteen different sites with typicalregistration accuracy to the synthetic image of less than a pixel in thebundle adjusted geometry. Since local pixel-by-pixel 3D knowledge is notused in embodiments, the accuracy of the fusion from the affinetransformation depends on the scene content and the disparity in theimaging perspectives of the two images.

Embodiments leverage precise registration and geometric adjustment of a2D image to 3D image registration as the geometry used to calculate thetransformation. It also uses the 3D point cloud to establish the bestelevation plane at which to create the affine that transforms the firstimage 100 to the space of the second image 200.

FIG. 9 illustrates, by way of example, a block diagram of an embodimentof a machine 900 on which one or more of the methods, such as thosediscussed about FIGS. 1-8 can be implemented. In one or moreembodiments, one or more operations of the method 300 can be implementedby the machine 900. In alternative embodiments, the machine 900 operatesas a standalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine 900 may operate in thecapacity of a server or a client machine in server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 900 may be a personal computer (PC), atablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), acellular telephone, a web appliance, a network router, switch or bridge,embedded computer or hardware, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methodologiesdiscussed herein.

The example machine 900 includes processing circuitry 1002 (e.g., ahardware processor, such as can include a central processing unit (CPU),a graphics processing unit (GPU), an application specific integratedcircuit, circuitry, such as one or more transistors, resistors,capacitors, inductors, diodes, logic gates, multiplexers, oscillators,buffers, modulators, regulators, amplifiers, demodulators, or radios(e.g., transmit circuitry or receive circuitry or transceiver circuitry,such as RF or other electromagnetic, optical, audio, non-audibleacoustic, or the like), sensors 1021 (e.g., a transducer that convertsone form of energy (e.g., light, heat, electrical, mechanical, or otherenergy) to another form of energy), or the like, or a combinationthereof), a main memory 1004 and a static memory 1006, which communicatewith each other and all other elements of machine 900 via a bus 1008.The transmit circuitry or receive circuitry can include one or moreantennas, oscillators, modulators, regulators, amplifiers, demodulators,optical receivers or transmitters, acoustic receivers (e.g.,microphones) or transmitters (e.g., speakers) or the like. The RFtransmit circuitry can be configured to produce energy at a specifiedprimary frequency to include a specified harmonic frequency.

The machine 900 (e.g., computer system) may further include a videodisplay unit 1010 (e.g., a liquid crystal display (LCD) or a cathode raytube (CRT)). The machine 900 also includes an alphanumeric input device1012 (e.g., a keyboard), a user interface (UI) navigation device 1014(e.g., a mouse), a disk drive or mass storage unit 1016, a signalgeneration device 1018 (e.g., a speaker) and a network interface device1020.

The mass storage unit 1016 includes a machine-readable medium 1022 onwhich is stored one or more sets of instructions and data structures(e.g., software) 1024 embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1024 mayalso reside, completely or at least partially, within the main memory1004 and/or within the processing circuitry 1002 during executionthereof by the machine 900, the main memory 1004 and the processingcircuitry 1002 also constituting machine-readable media. One or more ofthe main memory 1004, the mass storage unit 1016, or other memory devicecan store the job data, transmitter characteristics, or other data forexecuting the method 300.

The machine 900 as illustrated includes an output controller 1028. Theoutput controller 1028 manages data flow to/from the machine 900. Theoutput controller 1028 is sometimes called a device controller, withsoftware that directly interacts with the output controller 1028 beingcalled a device driver.

While the machine-readable medium 1022 is shown in an example embodimentto be a single medium, the term “machine-readable medium” may include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore instructions or data structures. The term “machine-readable medium”shall also be taken to include any tangible medium that can store,encode or carry instructions for execution by the machine and that causethe machine to perform any one or more of the methodologies of thepresent invention, or that can store, encode or carry data structuresutilized by or associated with such instructions. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, solid-state memories, and optical and magnetic media.Specific examples of machine-readable media include non-volatile memory,including by way of example semiconductor memory devices, e.g., ErasableProgrammable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1024 may further be transmitted or received over acommunications network 1026 using a transmission medium. Theinstructions 1024 may be transmitted using the network interface device1020 and any one of several well-known transfer protocols (e.g.,hypertext transfer protocol (HTTP), user datagram protocol (UDP),transmission control protocol (TCP)/internet protocol (IP)). The network1026 can include a point-to-point link using a serial protocol, or otherwell-known transfer protocol. Examples of communication networks includea local area network (“LAN”), a wide area network (“WAN”), the Internet,mobile telephone networks, Plain Old Telephone (POTS) networks, andwireless data networks (e.g., WiFi and WiMax networks). The term“transmission medium” shall be taken to include any intangible mediumthat can store, encode or carry instructions for execution by themachine, and includes digital or analog communications signals or otherintangible media to facilitate communication of such software.

EXAMPLES AND ADDITIONAL NOTES

Example 1 can include a method for fusing images, the method comprisingregistering a first image of a region to a three-dimensional (3D) pointset of the region to generate a registered first image, registering asecond image of the region to the 3D point set to generate a registeredsecond image, identifying, based on the 3D point set, geometric tiepoints of the registered first image and the registered second image,projecting, using an affine transformation determined based on theidentified geometric tie points, pixels of the registered first image toan image space of the registered second image to generate a registeredand transformed first image, and displaying the registered andtransformed first image and the registered second image simultaneously.

In Example 2, Example 1 can further include, wherein the first image isgenerated using a first optical device and the second image is generatedusing a second, different optical device.

In Example 3, at least one of Examples 1-2 can further include, whereinthe first image is of a first image type including one of anelectro-optical (EO) image, a synthetic aperture radar (SAR) image, aninfrared (IR) image, and a multi-spectral image, and the second image isof a second, different image type.

In Example 4, at least one of Examples 1-3 can further include, whereinthe geometric tie points are at substantially a same elevation based onthe 3D point set.

In Example 5, Example 4 can further include, wherein the geometric tiepoints are situated in a regular grid on the first and second images.

In Example 6, Example 5 can further include, wherein x,y points of thegrid are projected to the image space of the first image and the secondimage to generate geometric tie point coordinates between the images. InExample 7, Example 6 can further include generating the affinetransformation to project the geometric tie point coordinates in thefirst image to the geometric tie point coordinates in the second image.

In Example 8, at least one of Examples 1-7 can further includedetermining a corresponding pixel in the registered image to which atransformed projected first image pixel of the transformed projectedfirst image pixels corresponds using a bilinear interpolation.

Example 9 includes a system for fusing images, the system comprisingprocessing circuitry configured to registering a first image of a regionto a three-dimensional (3D) point set of the region to generate aregistered first image, registering a second image of the region to the3D point set to generate a registered second image, identifying, basedon the 3D point set, geometric tie points of the registered first imageand the registered second image, projecting, using an affinetransformation determined based on the identified geometric tie points,pixels of the registered first image to an image space of the registeredsecond image to generate a registered and transformed first image, and adisplay to provide a view of the registered and transformed first imageand the registered second image simultaneously.

In Example 10, Example 9 can further include, wherein the first image isgenerated using a first optical device and the second image is generatedusing a second, different optical device.

In Example 11, at least one of Examples 9-10 can further include,wherein the first image is of a first image type including one of anelectro-optical (EO) image, a synthetic aperture radar (SAR) image, aninfrared (IR) image, and a multi-spectral image, and the second image isof a second, different image type.

In Example 12, at least one of Examples 9-11 can further include,wherein the geometric tie points are at a same elevation determinedbased on the 3D point set.

In Example 13, Example 12 can further include, wherein the geometric tiepoints are situated in a regular grid on the first and second images.

In Example 14, Example 13 can further include, wherein the geometric tiepoints are formed from projecting a regular grid of x, y points at thesame elevation to the space of the first and second images.

In Example 15, Example 14 can further include, wherein the processingcircuitry is further configured to generate the affine transformation toproject the geometric tie point coordinates in the first image to thegeometric tie point coordinates in the second image.

In Example 16, at least one of Examples 9-15 can further includedetermining a corresponding pixel in the registered image to which atransformed projected first image pixel of the transformed projectedfirst image pixels corresponds using a bilinear interpolation.

Example 17 includes at least one non-transitory machine-readable mediumincluding instructions, that when executed by a machine, configure themachine to perform operations comprising registering a first image of aregion to a three-dimensional (3D) point set of the region to generate aregistered first image, registering a second image of the region to the3D point set to generate a registered second image, identifying, basedon the 3D point set, geometric tie points of the registered first imageand the registered second image, projecting, using an affinetransformation determined based on the identified geometric tie points,pixels of the registered first image to an image space of the registeredsecond image to generate a registered and transformed first image, andproviding signals that cause a display to provide a view of theregistered and transformed first image and the registered second imagesimultaneously.

In Example 18, Example 17 can further include, wherein the first imageis generated using a first optical device and the second image isgenerated using a second, different optical device.

In Example 19, at least one of Examples 17-18 can further include,wherein the first image is of a first image type including one of anelectro-optical (EO) image, a synthetic aperture radar (SAR) image, aninfrared (IR) image, and a multi-spectral image, and the second image isof a second, different image type.

In Example 20, at least one of Examples 17-19 can further include,wherein the geometric tie points are at a same elevation determinedbased on the 3D point set.

In Example 21, Example 20 can further include, wherein the geometric tiepoints are situated in a regular grid on the first and second images.

In Example 22, Example 21 can further include, wherein the geometric tiepoints are projected to the image space of the first image and thesecond image to generate geometric tie point coordinates between theimages.

In Example 23, Example 22 can further include, wherein the processingcircuitry is further configured to generate the affine transformation toproject the geometric tie point coordinates in the first image to thegeometric tie point coordinates in the second image.

In Example 24, at least one of Examples 17-23 can further includedetermining a corresponding pixel in the registered image to which atransformed projected first image pixel of the transformed projectedfirst image pixels corresponds using a bilinear interpolation.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

What is claimed is:
 1. A method for fusing images, the methodcomprising: adjusting, using ground control points (GCPs), imagegeometry in metadata of a two-dimensional (2D) first image of a regionto match image geometry of a three-dimensional (3D) point set of atleast a portion of the region to generate a registered first image, theGCS independent of the 3D point set; adjusting, using the GCPs, imagegeometry in metadata of a 2D second image of at least the portion of theregion to match the image geometry of the 3D point set to generate aregistered second image; identifying, based on the 3D point set,geometric tie points of the registered first image and the registeredsecond image; and projecting, using an affine transformation determinedbased on a least squares fit of the identified geometric tie points,pixels of the registered first image to an image space of the registeredsecond image to generate a registered and transformed first image. 2.The method of claim 1, further comprising displaying the registered andtransformed first image and the registered second image simultaneously.3. The method of claim 1, wherein the 2D first image is generated usinga first optical device and the 2D second image is generated using asecond, different optical device.
 4. The method of claim 1, wherein the2D first image is of a first image type including one of anelectro-optical (EO) image, a synthetic aperture radar (SAR) image, aninfrared (IR) image, and a multi-spectral image, and the 2D second imageis of a second, different image type.
 5. The method of claim 4, whereinthe geometric tie points are situated in a regular grid on theregistered first and second images.
 6. The method of claim 5, wherein x,y points of the grid are projected to the image space of the registeredfirst image and the registered second image to generate geometric tiepoint coordinates between the images.
 7. The method of claim 6, furthercomprising generating the affine transformation to project the geometrictie point coordinates in the registered first image to the geometric tiepoint coordinates in the registered second image.
 8. The method of claim1, wherein the geometric tie points are at substantially a sameelevation based on the 3D point set.
 9. The method of claim 1, furthercomprising determining a corresponding pixel in the registered image towhich a transformed projected first image pixel of the transformedprojected first image pixels corresponds using a bilinear interpolation.10. The method of claim 1, wherein the 2D first image and the 2D secondimage are captured from different perspectives.
 11. A system for fusingimages, the system comprising: processing circuitry configured to:adjust, using ground control points (GCPs), image geometry in metadataof a two-dimensional (2D) first image of a region to match imagegeometry of a three-dimensional (3D) point set of the region to generatea registered first image, the GCPs independent of the 3D point set;adjust, using the GCPs, image geometry in metadata of a 2D second imageof the region to match image geometry of the 3D point set to generate aregistered second image; identifying, based on the 3D point set,geometric tie points of the registered first image and the registeredsecond image; projecting, using an affine transformation determinedbased on a least squares fit of the identified geometric tie points,pixels of the registered first image to an image space of the registeredsecond image to generate a registered and transformed first image; and adisplay to provide a view of the registered and transformed first imageand the registered second image simultaneously.
 12. The system of claim11, wherein the 2D first image is generated using a first optical deviceand the 2D second image is generated using a second, different opticaldevice.
 13. The system of claim 11, wherein the 2D first image is of afirst image type including one of an electro-optical (EO) image, asynthetic aperture radar (SAR) image, an infrared (IR) image, and amulti-spectral image, and the 2D second image is of a second, differentimage type.
 14. The system of claim 11, wherein the geometric tie pointsare at a same elevation determined based on the 3D point set.
 15. Thesystem of claim 14, wherein the geometric tie points are situated in aregular grid on the registered first and second images.
 16. The systemof claim 15, wherein x, y points of the grid are projected to the imagespace of the registered first image and the registered second image togenerate geometric tie point coordinates between the images.
 17. Atleast one non-transitory machine-readable medium including instructions,that when executed by a machine, configure the machine to performoperations comprising: adjusting, using ground control points (GCPs),image geometry in metadata of a two-dimensional (2D) first image of aregion to match image geometry of a three-dimensional (3D) point set ofthe region to generate a registered first image, the GCPs independentfrom the 3D point set; adjusting, using the GCPs, image geometry inmetadata of a 2D second image of the region to match image geometry ofthe 3D point set to generate a registered second image; identifying,based on the 3D point set, geometric tie points of the registered firstimage and the registered second image; projecting, using an affinetransformation determined based on a least squares fit of the identifiedgeometric tie points, pixels of the registered first image to an imagespace of the registered second image to generate a registered andtransformed first image; and providing signals that cause a display toprovide a view of the registered and transformed first image and theregistered second image simultaneously.
 18. The at least onenon-transitory machine-readable medium of claim 17, wherein thegeometric tie points are at substantially a same elevation based on the3D point set and situated in a regular grid on the registered first andsecond images.
 19. The at least one non-transitory machine-readablemedium of claim 18, wherein x,y points of the grid are projected to theimage space of the registered first image and the registered secondimage to generate geometric tie point coordinates between the images,and the operations further include generating the affine transformationto project the geometric tie point coordinates in the registered firstimage to the geometric tie point coordinates in the registered secondimage.
 20. The at least one non-transitory machine-readable medium ofclaim 17, wherein the operations further comprise determining acorresponding pixel in the registered first image to which a transformedprojected first image pixel of the transformed projected first imagepixels corresponds using a bilinear interpolation.